Advanced CI
Part 2, Chapter 16
In this chapter, we'll update the CI process and set up a multistage Docker build for production.
Multistage Docker Build
Start by updating Dockerfile.prod like so:
###########
# BUILDER #
###########
# pull official base image
FROM python:3.10.1-slim-buster as builder
# install system dependencies
RUN apt-get update \
&& apt-get -y install gcc postgresql \
&& apt-get clean
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt
# lint
COPY . /usr/src/app/
RUN pip install black==21.12b0 flake8==4.0.1 isort==5.10.1
RUN flake8 .
RUN black --exclude=migrations .
RUN isort .
#########
# FINAL #
#########
# pull official base image
FROM python:3.10.1-slim-buster
# create directory for the app user
RUN mkdir -p /home/app
# create the app user
RUN addgroup --system app && adduser --system --group app
# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
WORKDIR $APP_HOME
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV ENVIRONMENT prod
ENV TESTING 0
# install system dependencies
RUN apt-get update \
&& apt-get -y install netcat gcc postgresql \
&& apt-get clean
# install python dependencies
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /usr/src/app/requirements.txt .
RUN pip install --upgrade pip
RUN pip install --no-cache /wheels/*
RUN pip install "uvicorn[standard]==0.16.0"
# add app
COPY . .
# chown all the files to the app user
RUN chown -R app:app $HOME
# change to the app user
USER app
# run gunicorn
CMD gunicorn --bind 0.0.0.0:$PORT app.main:app -k uvicorn.workers.UvicornWorker
Here, we used a Docker multistage build to reduce the final image size. Essentially, builder is a temporary image that's used for building the Python wheels. The wheels are then copied over to the final production image and the builder image is discarded.
Did you notice that Flake8, Black, and isort are being run in the builder image? If any of them fail, the build stops.
To test locally, build the new image and spin up the container:
$ docker build -f project/Dockerfile.prod -t web ./project
$ docker run --name fastapi-tdd -e PORT=8765 -e DATABASE_URL=sqlite://sqlite.db -p 5003:8765 web:latest
You should see something similar to:
[2021-12-28 00:30:21 +0000] [8] [INFO] Starting gunicorn 20.1.0
[2021-12-28 00:30:21 +0000] [8] [INFO] Listening at: http://0.0.0.0:8765 (8)
[2021-12-28 00:30:21 +0000] [8] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-12-28 00:30:21 +0000] [9] [INFO] Booting worker with pid: 9
[2021-12-28 00:30:22 +0000] [9] [INFO] Started server process [9]
[2021-12-28 00:30:22 +0000] [9] [INFO] Waiting for application startup.
[2021-12-28 00:30:22 +0000] [9] [INFO] Application startup complete.
Navigate to http://localhost:5003/ping/.
You should see:
{
"ping": "pong!",
"environment": "prod",
"testing": false
}
Bring down the container once done:
$ docker rm fastapi-tdd -f
Commit and push your code up to GitHub to trigger a new build. Ensure it passes.
Docker Caching
Next, to speed up the build on GitHub Actions, update the .github/workflows/main.yml file:
name: Continuous Integration and Delivery
on: [push]
env:
IMAGE: docker.pkg.github.com/$(echo $GITHUB_REPOSITORY | tr '[A-Z]' '[a-z]')/summarizer
jobs:
build:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2.4.0
with:
ref: main
- name: Log in to GitHub Packages
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Pull images
run: |
docker pull ${{ env.IMAGE }}-builder:latest || true
docker pull ${{ env.IMAGE }}-final:latest || true
- name: Build images
run: |
docker build \
--target builder \
--cache-from ${{ env.IMAGE }}-builder:latest \
--tag ${{ env.IMAGE }}-builder:latest \
--file ./project/Dockerfile.prod \
"./project"
docker build \
--cache-from ${{ env.IMAGE }}-final:latest \
--tag ${{ env.IMAGE }}-final:latest \
--file ./project/Dockerfile.prod \
"./project"
- name: Push images
run: |
docker push ${{ env.IMAGE }}-builder:latest
docker push ${{ env.IMAGE }}-final:latest
test:
name: Test Docker Image
runs-on: ubuntu-latest
needs: build
steps:
- name: Checkout
uses: actions/checkout@v2.4.0
with:
ref: main
- name: Log in to GitHub Packages
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Pull images
run: |
docker pull ${{ env.IMAGE }}-builder:latest || true
docker pull ${{ env.IMAGE }}-final:latest || true
- name: Build images
run: |
docker build \
--target builder \
--cache-from ${{ env.IMAGE }}-builder:latest \
--tag ${{ env.IMAGE }}-builder:latest \
--file ./project/Dockerfile.prod \
"./project"
docker build \
--cache-from ${{ env.IMAGE }}-final:latest \
--tag ${{ env.IMAGE }}-final:latest \
--file ./project/Dockerfile.prod \
"./project"
- name: Run container
run: |
docker run \
-d \
--name fastapi-tdd \
-e PORT=8765 \
-e ENVIRONMENT=dev \
-e DATABASE_URL=sqlite://sqlite.db \
-e DATABASE_TEST_URL=sqlite://sqlite.db \
-p 5003:8765 \
${{ env.IMAGE }}-final:latest
- name: Pytest
run: docker exec fastapi-tdd python -m pytest .
- name: Flake8
run: docker exec fastapi-tdd python -m flake8 .
- name: Black
run: docker exec fastapi-tdd python -m black . --check
- name: isort
run: docker exec fastapi-tdd python -m isort . --check-only
deploy:
name: Deploy to Heroku
runs-on: ubuntu-latest
needs: [build, test]
env:
HEROKU_APP_NAME: sleepy-thicket-77414
HEROKU_REGISTRY_IMAGE: registry.heroku.com/${HEROKU_APP_NAME}/summarizer
steps:
- name: Checkout
uses: actions/checkout@v2.4.0
with:
ref: main
- name: Log in to GitHub Packages
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Pull images
run: |
docker pull ${{ env.IMAGE }}-builder:latest || true
docker pull ${{ env.IMAGE }}-final:latest || true
- name: Build images
run: |
docker build \
--target builder \
--cache-from ${{ env.IMAGE }}-builder:latest \
--tag ${{ env.IMAGE }}-builder:latest \
--file ./project/Dockerfile.prod \
"./project"
docker build \
--cache-from ${{ env.IMAGE }}-final:latest \
--tag ${{ env.IMAGE }}:latest \
--tag ${{ env.HEROKU_REGISTRY_IMAGE }}:latest \
--file ./project/Dockerfile.prod \
"./project"
- name: Log in to the Heroku Container Registry
run: docker login -u _ -p ${HEROKU_AUTH_TOKEN} registry.heroku.com
env:
HEROKU_AUTH_TOKEN: ${{ secrets.HEROKU_AUTH_TOKEN }}
- name: Push to the registry
run: docker push ${{ env.HEROKU_REGISTRY_IMAGE }}:latest
- name: Set environment variables
run: |
echo "HEROKU_REGISTRY_IMAGE=${{ env.HEROKU_REGISTRY_IMAGE }}" >> $GITHUB_ENV
echo "HEROKU_AUTH_TOKEN=${{ secrets.HEROKU_AUTH_TOKEN }}" >> $GITHUB_ENV
- name: Release
run: |
chmod +x ./release.sh
./release.sh
We're now pulling, building, and pushing the builder image using the --target option.
For more on Docker caching, review the Faster CI Builds with Docker Cache article.
Update Requirements
Next, let's move development-only requirements to a new file.
requirements-dev.txt:
black==21.12b0
flake8==4.0.1
isort==5.10.1
pytest==6.2.5
pytest-cov==3.0.0
pytest-xdist==2.5.0
-r requirements.txt
requirements.txt:
aerich==0.6.1
asyncpg==0.25.0
fastapi==0.70.1
gunicorn==20.1.0
newspaper3k==0.2.8
requests==2.26.0
tortoise-orm==0.17.8
uvicorn==0.16.0
Update Dockerfile:
# pull official base image
FROM python:3.10.1-slim-buster
# set working directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install system dependencies
RUN apt-get update \
&& apt-get -y install netcat gcc postgresql \
&& apt-get clean
# install python dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
COPY ./requirements-dev.txt .
RUN pip install -r requirements-dev.txt
# add app
COPY . .
# add entrypoint.sh
COPY ./entrypoint.sh .
RUN chmod +x /usr/src/app/entrypoint.sh
# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]
Test:
$ docker-compose down -v
$ docker-compose up -d --build
$ docker-compose exec web aerich upgrade
$ docker-compose exec web python -m pytest
$ docker-compose exec web flake8 .
$ docker-compose exec web black .
$ docker-compose exec web isort .
Install the requirements in the test job of .github/workflows/main.yml:
- name: Install requirements
run: docker exec fastapi-tdd pip install black==21.12b0 flake8==4.0.1 isort==5.10.1 pytest==6.2.5
Full job:
test:
name: Test Docker Image
runs-on: ubuntu-latest
needs: build
steps:
- name: Checkout
uses: actions/checkout@v2.4.0
with:
ref: main
- name: Log in to GitHub Packages
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Pull images
run: |
docker pull ${{ env.IMAGE }}-builder:latest || true
docker pull ${{ env.IMAGE }}-final:latest || true
- name: Build images
run: |
docker build \
--target builder \
--cache-from ${{ env.IMAGE }}-builder:latest \
--tag ${{ env.IMAGE }}-builder:latest \
--file ./project/Dockerfile.prod \
"./project"
docker build \
--cache-from ${{ env.IMAGE }}-final:latest \
--tag ${{ env.IMAGE }}-final:latest \
--file ./project/Dockerfile.prod \
"./project"
- name: Run container
run: |
docker run \
-d \
--name fastapi-tdd \
-e PORT=8765 \
-e ENVIRONMENT=dev \
-e DATABASE_URL=sqlite://sqlite.db \
-e DATABASE_TEST_URL=sqlite://sqlite.db \
-p 5003:8765 \
${{ env.IMAGE }}-final:latest
- name: Install requirements
run: docker exec fastapi-tdd pip install black==21.12b0 flake8==4.0.1 isort==5.10.1 pytest==6.2.5
- name: Pytest
run: docker exec fastapi-tdd python -m pytest .
- name: Flake8
run: docker exec fastapi-tdd python -m flake8 .
- name: Black
run: docker exec fastapi-tdd python -m black . --check
- name: isort
run: docker exec fastapi-tdd python -m isort . --check-only
Again, commit and push your code up to GitHub to trigger a new build. Make sure it passes before moving on.
✓ Mark as Completed